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I. 


INTRODUCTION 


A. INTRODUCTION 

The volume of services, both in dollar amount and type, acquired by the 
Department of Defense (DoD) has steadily grown over the past few fiscal years. In the 
period between fiscal years 2000 and 2005, DoD experienced a 73-percent increase in 
service contract costs in the operations and maintenance areas. The chief factors behind 
the growth in acquired services include 1) the impact the Global War on Terrorism has 
had on the number of requirements and the resulting increase in the DoD’s use of 
contractors to meet those needs, 2) the federal policy mandating reliance on contractors 
for functions not inherently governmental, 3) competitive sourcing and privatization 
programs within the DoD and 4) headquarters personnel manning level limitations 
leading to dependence on contractors to complete new and expanded work requirements. 
(GAO, 2007) 

To achieve increased data granularity on these primary factors and an accurate 
weighting of each one’s impact, empirical data needs to be collected and analyzed. We 
believe that data collection at the installation level throughout DoD will provide the 
necessary granularity. Coding and analysis of this gathered data should clarify whether 
the recent trends will continue or abate and whether they are “treatable” or inherent to 
modem day DoD services acquisition. Once the ground truth of the current services 
acquisition situation is better understood, alternatives could be developed, and sound 
services acquisition management policy recommendations can be forwarded to DoD 
leadership for consideration and implementation. 

Directly linked with the acquisition of services is the management of those 

acquired services. Apte, Ferrer, Lewis, and Rendon (2006) conducted exploratory 

research regarding the size, trends, and issues related to services acquisition. Data was 

collected through site visits to the Presidio of Monterey, Travis Air Force Base, and the 

Naval Postgraduate School. One of their research findings was that despite the high 

expenditure levels for services, the acquisition management stmcture for service 
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contracts is lacking. Unlike the management infrastructure for a large weapon system 
procurement program, which would have a dedicated program manager and robust 
management team, services acquisition procurement programs do not offer the same level 
of oversight, thus opening the door for mismanagement. 

Apte and Rendon (2007) conducted a follow-up research project with a 
concentration on the applicability of a program management approach in managing 
acquired services within DoD. The project provided further discussion of issues in 
services contracting including the intangibility of service outcomes, co-production of 
services and the diversity of services. In their research the principal investigators 
discussed basic concepts of program management and how those concepts are currently 
applied to the acquisition of products. Using this framework, the project analyzed 
services acquisition management at the installations they visited in the first project, in 
addition to Randolph Air Force Base in Texas. It was concluded that though some 
program management concepts were being applied, they were being applied 
inconsistently and did not necessarily build the framework for a program management 
approach to services acquisition. 

B. PURPOSE 

The purpose of this project is to determine how to best collect empirical data 
regarding the current state of services acquisition management at the installation level 
across the military services and to conduct an initial analysis of collected data. The 
results of this project will support on-going research (Apte et ah, 2006, Apte and Rendon, 
2007) being conducted by the Acquisition Research Program at the Naval Postgraduate 
School regarding the Department of Defense’s management of services acquisition. 

C. RESEARCH QUESTIONS 

There are several research questions which this project tries to answer and they 
are as follows. 


2 



Primary: 

• What is the current state of services acquisition management, at the installation 
level? 

Secondary: 

• What research method should be used to best evaluate the current state of services 
acquisition management, across all military services, in a uniform and unbiased 
manner? 

• What is the best way to tailor the chosen research method to produce usable 
results? 

• How should the chosen research method be tailored to answer the primary 
research question? 

Researching answers to these questions will provide information and data will 
facilitate in studying the six research questions posed by the aforementioned of on-going 
research. These six research questions are 1) What types of services are typically 
contracted for at military installations and what is the annual expenditure for these 
services? 2) What types of acquisition strategies, procurement methods, and contracts are 
being used to acquire services? 3) How are these service contracts managed? 4) What 
types of organization/management structures are used to manage contracted services?; 5) 
What training does contract and project/program management staff receive? and 6) Do 
the respective military services acquire and manage services differently? 

D. SERVICE CATEGORIES 

The Federal Procurement Data System Product and Service Codes Manual 
identifies and describes twenty-four service categories used in the Federal Procurement 
Data System. The types of services provided for under these categories range from 
special studies and analysis to utilities and housekeeping services. Table 1 on the 
following page provides the complete listing of service categories and associated PSC 
codes. 
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Table 1. Product Service Categories and Codes 


Service Category 

Product/Service 
Classification (PSC) Code 

Research and development 

A 

Special studies and analysis 

B 

Architect and engineering services 

C 

Data processing and telecommunications 

D 

Purchase of structures and facilities 

E 

Conservation and natural resources 

L 

Social Services 

G 

Quality control, testing and inspection services 

H 

Maintenance and repair of equipment 

J 

Modification of equipment 

K 

Technical representative services 

L 

Operation of government-owned facilities 

M 

Installation of equipment 

N 

Salvage services 

P 

Medical 

Q 

Professional, administrative, and management support 

R 

Utilities and housekeeping 

S 

Photographic, mapping, printing and publication 
services 

T 

Educational and training services 

U 

Transportation and travel 

V 

Lease or rental of equipment 

w 

Lease or rental of facilities 

X 

Construction of structures and facilities 

Y 

Maintenance and repair of real property 

Z 


E. PROJECT SCOPE LIMITATIONS 

The shear volume of services acquisition within the DoD along with other 
limiting factors decided the scope of this project. The first is the reduction in the number 
of service categories considered to seven of the twenty-four mentioned in Table 1. These 
seven service categories were selected for further examination based on data presented in 
the Government Accountability Office’s (GAO) May 2007 report, Defense Budget: 
Trends in Operation and Maintenance Costs and Support Services Contracting as shown 
in Table 2. 
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Table 2. Changes in Service Contract Costs in Selected Categories (From 
GAO 2007) 


Fiscal year 2007 dollars in billions 


Contract Costs 

Change from fiscal 
year 2000 to 2005 


FY 

FY 



Service category 

2000 

2005 Amount Percentage 

Professional, administrative, and management support 

$14.6 

$30.1 

$15.5 

107 

Maintenance and repair of equipment 

7.7 

12.3 

4.6 

60 

Data processing and telecommunications 

6.3 

11.0 

4.7 

74 

Medical 

2.8 

8.4 

5.6 

199 

Maintenance and repair of real property 

6.6 

8.0 

1.5 

22 

Utilities and housekeeping 

3.9 

7.0 

3.1 

79 

Transportation and travel 

3.4 

6.6 

3.3 

97 

Conservation and natural resources 

1.7 

2.3 

0.7 

39 

Operation of government-owned facilities 

2.3 

2.1 

(0.2) 

(9) 

Technical representative services 

1.4 

1.7 

0.3 

23 

Special studies and analyses 

1.2 

1.5 

0.2 

19 

Modification of equipment 

1.1 

1.4 

0.3 

29 

Educational and training services 

1.1 

1.4 

0.3 

23 

Other 

1.3 

2.0 

0.7 

58 

Total 

$55.4 

$95.9 

$40.6 

73 


GAO considered 19 of the 24 service categories in the report. The seven service 
categories selected for consideration within this project, highlighted in Table 3 on the 
next page, accounted for more than $83 billion in expenditures during fiscal year 2005, 
up from $45.3 billion in fiscal year 2000. The $83 billion spent on services in fiscal year 
2005 accounted for roughly 87-percent of expenditures on services. Additionally, six of 
the seven selected categories of services showed the greatest percentage changed in 
dollars spent between fiscal year 2000 and fiscal year 2005. 
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Table 3. Service Categories Considered by Project 


Service Category 

T-^ 

Product/Service f 

Classification (PSC) Code I, 

Professional, administrative, and management support 

R 

Maintenance and repair of equipment 

J 

Data processing and telecommunications 

D 

Medical 

Q 

Utilities and housekeeping 

S 

Transportation and travel 

V 

Maintenance and repair of real property 

z 


We further limited our research to installations within the continental United 
States (CONUS) and eliminated international bases. This limitation arose from 
complications in accounting for data considering varying currency exchange rates, 
economies and operational situations indicative of divergent foreign locations, not to 
mention communications across time zones. 

Another limitation is the number of military services to consider. Research is 
needed across all four branches of military service. However with limited personnel 
resources and the project team members’ respective affiliation with the Army and Marine 
Corps, this project is limited to those military services’ installations. All four services 
would be beyond the current capabilities of the project team, but follow-on projects to 
capture and analyze Air Force and Navy installations data are already underway. 

F. METHODOLOGY 

A survey methodology will be used to facilitate the gathering of preliminary data. 
A survey will be designed and issued to a sample audience for conduct of the pilot test. 
The qualitative results of the pilot test will be used to adjust the survey to increase survey 
reliability. The quantitative results of the pilot test will be analyzed and preliminary 
observations regarding the acquisition management of services will be provided. 
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G. ORGANIZATION 

This project is organized into five chapters. This introductory chapter is followed 
by Chapter II which describes the aspects of survey methodology. Chapter III is the 
presentation and discussion of the survey created by this project using the survey 
methodology discussed in Chapter II. Chapter IV is an initial analysis of data collected 
during the survey pilot test and recommendations for improving the survey for follow-on 
research projects. Chapter V concludes with the summary and implications of the 
analysis and recommendations for further study in the area of the management of services 
acquisition within the Department of Defense. 
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II. SURVEY METHODOLOGY 


A. INTRODUCTION 

Surveys of all types have been and continue to be widely used in virtually all 
areas of public policy. In the United States the most notable survey is the decennial 
census conducted by the U.S. Census Bureau. In addition to conducting the decennial 
census, the U.S. Census Bureau conducts over 100 other survey every year (U.S. Census 
Bureau, 2007c). The other surveys carried out by the U.S. Census Bureau focus on 
numerous areas focused on demographic or economic characteristics. Demographic 
surveys are conducted pertaining to areas such as housing; fishing, hunting, and wildlife- 
associated recreation; property owners and managers; income; and women- and minority- 
owned businesses (U.S. Census Bureau, 2007a). Economic surveys are conducted 
regarding areas such as construction, foreign trade, manufacturing, mining, retail, 
wholesale and services (U.S. Census Bureau, 2007b). The purpose of these surveys is to 
collect generic statistical information from individuals and establishments within the 
respective categories and use the resulting statistics to shape public policy. (U.S. Census 
Bureau, 2007c). 

There are numerous types of surveys, each with a different purpose. Despite the 
variability in survey types, each survey should follow some basic design steps and 
implement measures to protect respondents. The basic steps in designing a survey are 1) 
define the objectives for the survey (Fowler, Jr., 1984), 2) determine how often the 
survey will be administered, 3) generate questions and completion instructions for 
respondents, 4) plan for how the data will be analyzed, 5) conduct a pilot test, and 6) 
report the results. (Fink, 2006) 

B. TYPES OF SURVEYS 

Surveys are just one method of collecting information to help describe, compare 
or explain various levels of knowledge both individual knowledge and/or societal 
knowledge. There are two basic types of surveys—interviews and self-administered. 
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Surveys can take one of a couple formats within each of these types. An interview type 
of survey can present the questions of the survey to the respondent via either the 
telephone or in-person. The respondents answers are recorded and analyzed at a later 
dater (Fink, 2006) 

Self-administered surveys on the other hand can be mailed to the respondent for 
completion and then returned to the administering authority. This format of survey can 
be conducted on-site at a centralized location, such as a public health clinic. Increasingly 
self-administered surveys are being conducted on-line. The underlying characteristic of 
self-administered surveys is that the respondent is able to complete the survey alone and 
in private or with the assistance of another person, other than an interviewer. (Fink, 
2006) 

The project team elected to create a survey as this would be the best method for 
gathering the necessary information. The mode of data collection is a web-based, self- 
administered type of survey in order to collect data to support this and follow-on projects. 
The necessity of utilizing a survey was to 1) standardize measurement of responses, and 
2) attain a greater level of granularity than that currently available in government-wide 
level reports (Fowler, Jr., 1984). Additional justifications for this survey format were the 
cost savings by 1) not having to mail the survey and postage-paid return envelopes to 
over 100 Army and Marine Corps CONUS installation and 2) the existence of a corporate 
account with the web-based survey engine SurveyMonkey at the Naval Postgraduate 
School. 


C. ADVANTAGES AND DISADVANTAGES 

Each survey type brings with it its own characteristics, advantages and 
disadvantages. Table 4 outlines a comparison of the various survey types discussed 
above. The table highlights the basic characteristic of each survey type, the advantages 
and disadvantages of each, along with special needs considerations and cost elements for 
each type. 
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Table 4. Survey comparison (From Fink, 2006) 


1 Self-Administered | 


Mailed 

On-stie 

Online 

Characteristics 

Paper and pencil 

Paper and pencil 

Internet based 


Can reach large 
geographic areas 

Information is 

obtained 

immediately 

Worldwide 


People are used to 
completing paper 
and pencil surveys 

Questions about 
survey can be asked 
by respondents as 
they arise 

Order of question 
can be programmed 

Advantages 

Can take the survey 
with you and 
complete it 
anywhere 

In some cases, 
surveys can be done 
with groups of 
people 

Only "legal" 

answeres are 

accepted 




Can give 
respondents links 
that explain 
unfamiliar words 
and help with 
difficult questions 




Data are 
automatically 
entered and be 
automatically 
analyzed 


Need a motivated 
sample to return 
survey. Many 
people think they 
have too much to do 
without also having 
to complete surveys 

Limited to responses 
from just those who 
are on site 

Need reliable access 

to Internet 

Disadvantages 

Respondents must 
be able to read, see, 
and write 

Respondents must 
be able to read, see, 
and write 

Respondent must be 
able to use a 

browser 




Browser must 

suppot survey 
graphics 




System can go down 
or be unreliable 


Up-to-date address 
list 

Space and privacy 
for respondent to 
complete the survey 

Technical expertise 

Special needs 

Follow-up mailings 


Convincing method 
of ensuring privacy 
and confientiality 


Incentives 







Costs 

Printing, paper, 
envelopes, stamps, 
incentives 

Printing, paper, 
incentives, survey 
supervisor, and 
possibly space for 
respondent to work 

Mainly technical 
(e.g., someone who 
is experience in 
designing online 
surveys) 


Interviews 


Telephone 

In-person 

Can be done with 
written script or 
computer assisted 

Can be doen with a 
written script or 
computer assisted 


Can explore answers Same as telephone 
with respondents 

Can assist 
respondent with 
unfamiliar words 



Need trained 

interviewers 

Need trained 

interviewers 

Need to make sure 
respondent is home 

If using computer- 
assisted interviews, 
will need technical 
expertise to program 
them 

Must find a suitable 
place to conduct 
interview 



Up-to-date phone 
numbers 

If on-site, need 
space and privacy 

Schedule for 

reaching 

respondents 

May be difficult or 
dangerous to go to 
person's home 

Many need a 
sampling expert for 
random digit dialing 


Incentives 


Training, incentives, 
telephones and 
telephone charges, 
computers and 
technical expertise, 
sampling expert, 
incentives 

Trainging, space, 
travel, incentives 
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Advantages of Web-based, Self-administered Questionnaire 


A web-based, self-administered questionnaire has several benefits over other 
survey research methods. The first benefit of the selected survey format is the ease of 
presentation with visual and or audio aids (Fowler, Jr., 1984). Survey designers with the 
use of computers may incorporate visual and audio aids to further assist the survey 
respondent in understanding terms or questions they don’t fully understand (Fink, 2006). 
A second benefit is the potential to ask questions with lengthy or complex response 
choices. Self-administered surveys allow the respondents to take their time to read each 
question and consider all of the possible choices without feeling the pressure to provide a 
response to a person. An additional benefit is the ease of asking numerous questions that 
are similar. A fourth benefit of using a self-administered questionnaire is the respondent 
does not have to feel uncomfortable in providing answers or opinions to another human 
in face-to-face contact such as is present with in-person interviews. (Fowler, Jr., 1984) 

2. Disadvantages of Web-based, Self-administered Questionnaire 

The election of a web-based, self-administered questionnaire does bring several 
disadvantages as well. The first is the level of care and attention demanded by question 
design. Without sound question design the respondent may not respond to the question 
or may provide an inaccurate response due to personal interpretation. Another 
disadvantage to this type of survey is the lack of having a researcher on site to supervise 
the completion of the survey, answer any questions respondents may have, or the quality 
of responses given. (Fowler, Jr., 1984). A third disadvantage is in direct contrast to the 
advantage of being able to incorporate visual and audio aids in the presentation of the 
survey. If the survey design team incorporates visual and audio aids risk comes in due to 
not all respondents having access to the same software programs or computer hardware 
(Fink, 2006). 

D. PILOT TEST 

Arguably the most important aspect of survey design is the conduct of a pilot test. 
A pilot test is the trial run of issuing the survey and collecting response data (Fink, 2006). 
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One of the key survey aspects being tested during this crucial methodology step is to 
ensure questions are written well. In addition the pilot test is a trial run of the 
appropriateness and applicability of survey questions in relation to the sample audience. 
Closely linked to ensuring the clarity and quality of questions is to verify if the collected 
data will provide the necessary information. Additionally, the pilot test provides insight 
into how consistent the collected data will be. (Fink, 2006). A properly conducted pilot 
test will prove invaluable in carrying out a reliable and valid survey to the sample 
audience. 

Receiving accurate and sufficient information about the survey characteristics 
being examined during a pilot test relies heavily on the response rate. The response rate 
is the percentage or portion of completed surveys received with respect to the total 
surveys issued (Fink, 2006). The recommended minimal response rate is between 50 and 
60 percent (Babbie, 2001 as referenced in Ruane, 2005). 

Figure 1. Response Rate Formula. 

„ Number of completed surveys received ^ 

Response rate =- - --*100% 

Number of surveys issued 


Self-administered surveys typically have lower preliminary response rates—less 
than 30 percent (Ruane, 2005). Some potential reasons for a low response rate are 1) the 
survey does not reach participants and therefore they cannot complete it, 2) participants 
refuse to complete the survey, and 3) participants are unable to answer the survey due to 
illness or language barriers (Fink, 2006). 

E. RELIABILITY AND VALIDITY 

Reliability and validity are very important intertwined characteristics for a survey. 
Without either of theses key characteristics, the work that went into designing and 
performing a survey may have been for naught. A valid survey will always be a reliable 
one but that same reliable survey may not always be valid for every instance (Fink, 
2006). 
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An example of this link between reliability and validity is provided in the 
following scenario. A hospital administrator is consistently asked how many patient beds 
are in a certain ward of the hospital. If the administrator provides the same answer each 
time the survey is administered, the survey is said to be reliable. However, if the 
researchers and survey presenters claim the same survey question provides information as 
to the level of quality of medical care, the validity of the survey would then be called into 
question. (Fink, 2006) 

1. Reliability 

A survey necessity, regardless the format, is the characteristic of being reliable. A 
reliable survey is one that provides consistent measures of important characteristics 
despite underlying changes in the target audience. Underlying changes in the target 
audience refer to changes in experiences, restfulness, anger, and tension at each 
respective time the survey was completed by respondents. (Fink, 2006) There are 
multiple means to check for reliability. 

a. Multiple Forms 

The multiple-form means of assessing reliability should be utilized when 
the principal investigators will only have access to the target audience once. Under this 
method information pertaining to a measure, such as a respondent’s age, is obtained by 
asking two separate questions. One of the questions regarding age may ask the 
respondent to provide their age. The other question might ask the respondent’s year of 
birth. If the answers to these individual questions are consistent, then reliability of the 
measure is achieved. (Ruane, 2005) 

b. Split-half Technique 

The split-half technique is used when the measure being examined is 
composed of multiple aspects. This technique is a means of inspecting the individual 
aspects and if the individual aspects consistently and equally contribute to the composite 
measure. The implementation of this method is with a battery of questions in which one 
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half of the list is considered as a mini-list and the other half is considered as another 
mini-list. If a comparison of the two mini-lists yields a high correlation, the entire list of 
questions is said to be reliable for assessing the measure. 

2. Validity 

Just as there were multiple means of checking for reliability, there are multiple 
means of ensuring validity. There are four validity techniques—face, content, criterion 
and construct—survey designers can use to establish and test for validity in their survey 
measures (Ruane, 2005). Each of these respective validity techniques is discussed below. 

a. Face Validity 

Face validity techniques simply ask if the measure appears to be okay or 
sound okay. Face validity assessment is subjective in nature and therefore can sometimes 
be called into question. An example of questionable face validity is connecting a fear of 
crime to a question that simply asks about one’s feeling towards walking at night alone 
(Ruane, 2005) 


b. Content Validity 

Content validity is important when the topic’s definition is multifaceted 
and complex. Content validity is a subjective assessment made by the researcher whether 
the measurement captures all of the facets of the complex problem. (Ruane, 2005) Often 
times it is difficult to capture the full meaning of the multifaceted and complex problem 
with just a single measurement or question. 

c. Criterion Validity 

Criterion validity is not a subjective assessment of a measure but rather 
uses empirical, objective data to explain the measure’s validity. There are two dominant 
strategies to use when checking for criterion validity—predictive and concurrent. If 
using predictive validity the principal investigators show the measurements validity when 
the measurement accurately forecasts a related outcome. Concurrent validity on the other 
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hand attempts to prove one measure’s validity by obtaining similar measurements with 
different tests of the same eoneept. (Ruane, 2005) 

d. Construct Validity 

Construet validity is probably the most diffieult of validity eheeks and 
involves theory and hypothesis testing. Theories are used to ereate hypotheses whieh 
prediet expected relationships between a measure and other variables. To claim construct 
validity support for the hypothesis has to be shown. Construct validity demands many 
man-hours and can be smaller, stand-alone research projects in and of themselves. 
(Ruane, 2005) 

F. ETHICS, PRIVACY, AND CONFIDENTIALITY 

A basic ethics guideline in all research involving human participants is the 
implementation of various measures to help ensure that no harm comes to any participant 
(Fowler, Jr., 1984). In addition to this “no brainer” ethical guideline, additional measures 
which must be implemented by researchers conducting research involving human 
subjects at agencies which receive federal funding are outlined in Part 46, Title 45 of the 
Code of Federal Regulations. Some of these additional measures include attaining 
informed consent from participants and approval from an Institutional Review Board 
(DHHS). 

I. Informed Consent 

Informed consent refers to the participant determining themselves whether to 
participate in the research or not (Ruane, 2005). Ruane (2005) provides a sound 
discussion of four characteristics of informed consent and they are competence—the 
ability of the participant to decide whether to participate or not; voluntarism—the 
participant’s ability to freely decide, without coercion or threat of retribution, to 
participate or not; full information—the right of the participant to be fully informed of all 
aspects of the research; and comprehension—the participant’s ability to understand all 
information given to them. 
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The Code of Federal Regulations lists eight general requirements for informed 
consent. Each of these general requirements needs to be presented to the perspective 
participant so that they can consider all factors in their decision to participate or not. The 
eight general requirements are 1) a statement that the project involves research, the 
purpose(s) of the research, anticipated duration of participation, a description of the 
procedures, even experimental ones, that are being followed, 2) description of foreseeable 
risks, 3) description of benefits to the participant, 4) procedures or other courses of 
treatment that might benefit the participant more, 5) extent of measures to maintain 
participant confidentiality, 6) description of compensation for research involving more 
than minimal risk to the participant, 7) provide points of contact for answering questions 
or addressing concerns, and 8) a statement that participation in the research is purely 
voluntary. (DHHS) 

2. Institutional Review Board (IRB) 

The Institutional Review Board (IRB) concept is in place to further provide 
protection to potential human research subjects. The implementation of an (IRB) is 
mandated by the Code of Federal Regulation for all agencies conducting research and in 
receipt of federal funding. The simplified purpose of an IRB is to review research 
proposals to ensure that the researchers have incorporated the other mandatory 
stipulations of 45 CFR 46, such as all the elements of informed consent being provided to 
each participant. The IRB does possess the authority to waive certain aspects of 45 CFR 
46 such as the researchers receiving a signed informed consent form from each 
participant when it can be demonstrated that the research effort could not practicably be 
completed without the requirement waiver. 

An IRB is granted certain authorities by 45 CFR 46. The first of which is the 
authority approve research proposals when there is a majority amongst the board 
members. In addition to approving research proposals, the IRB is may mandate certain 
changes to the research proposal prior to granting their approval. Once approved the 
researchers may not deviate from the approved protocol. If changes to the protocol need 
to be made, the principal investigators need to submit an updated protocol to the IRB for 
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consideration. In a case where an IRB disapproves a research proposal, a reviewing 
authority may not then approve the research. The IRB’s decision is final. (DHHS) 
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III. SERVICES ACQUISITION MANAGEMENT SURVEY 

DESIGN 


A. INTRODUCTION 

In the last chapter a general overview of survey research methodology was 
provided. Based on the advantages and disadvantages of various survey types the project 
team elected to conduct a web-based, self-administered questionnaire in order to gather 
data regarding the current state of services acquisition management, at the installation 
level, for the Army and Marine Corps. This chapter presents the methods, justifications 
and logic that were used to create the web-based, self-administered questionnaire within 
the framework of survey research methodology. 

B. QUESTIONNAIRE DESIGN 

The questionnaire was designed to collect empirical data to stratify the 
participants by military branch of service, region within that service, as applicable, and 
by the particular installation. The steps taken in designing this survey are the same as 
discussed in Chapter II and they are 1) define the objectives, 2) determine how often the 
survey will be administered, 3) generate questions and completion instructions for 
respondents, 4) plan for how the data will be analyzed, 5) conduct a pilot test, and 6) 
report the results. 

I. Objectives Defined 

The objectives of the survey are directly linked to our primary research question: 
What is the current state of services acquisition management at the installation level? In 
order to gain insight into the current state of services acquisition management the survey 
first needs to collect data and information on the acquisition of services within the seven 
selected service categories. In addition to providing information on the types of services 
contracted the survey needs to gather information on the acquisition strategies and 
procurement methods used to acquire the services. The third objective of the survey is to 
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provide insight into what organizational and management structures are used and how do 
they manage the contracts for services within the selected service categories. The final 
objective of the survey is to provide insight into what types of training the acquisition and 
project/program management team receives at each installation. 

2. Survey Duration 

This survey was designed to be issued once in order to attain a cross-sectional 
snapshot of services acquisition management for each of the services. Even though this 
project only covers Army and Marine Corps CONUS installations, follow-on research 
projects would use this same survey to gain data and information for the other military 
services. By using the same core survey the follow-on projects will be able to equally 
compare their results with the results of this project, therefore offering greater insight to 
other researchers as to the acquisition management of services within DoD. 

3. Informed Consent and Completion Instructions 

A key component to conducting a survey is providing the required information 
about the research project to potential respondent so they can make an informed consent 
statement. In designing our survey with the use of SurveyMonkey our first page is the 
“Participation Statement.” In this area the potential respondent is informed that the 
survey is part of a research project to help identify trends and best practices pertinent to 
the acquisition of services at the installation level. Respondents are also informed that 
the survey is completely voluntary and are provided with an estimated involvement time. 
A key component to this area is the conveyance that confidentiality and privacy will be 
maintained by not only the project team but the institution as well. Lastly, the respondent 
is provided contact information for the project team and the IRB so that their questions 
and concerns could be handled efficiently and effectively. Including all of the 
information in the Participation Statement section the team maintained compliance with 
45 CFR 46. 

Once the respondent elects to participate in the survey the next webpage seen is 
the “Introduction.” On this webpage the purpose of the project is again conveyed to the 
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participant along with the survey’s completion instructions. Participants are instructed to 
answer the questions to the best of their ability and are again informed that no personal 
information will be asked or recorded during the conduct of the survey. Information on 
how to move back and forth throughout the web-based questionnaire is also provided. 
One of the key features of using a commercial survey engine is the respondent does not 
need to complete the survey in one sitting. If the respondent needs to leave the survey 
site for any reason, the responses already selected will be saved and the survey will 
continue from the point where the respondent exited the website. Even though responses 
are saved for the respondents, no response information is forwarded to the researchers 
until the respondent selects the “Done” button following the final question of the survey. 

4. Question Design 

The questions of the survey are organized into one of four groups— 
administrative, core, general and comments—for both ease of design and flow for the 
respondent. The administrative group of questions is presented to the respondent first 
which helps identify the military service, geographical region, and individual installation 
being represented by the participant. In this and other sections of the web-based 
questionnaire, the project team utilized filter questions and a “skip-logic” tool provided 
by SurveyMonkey, whereby the participant’s responses to earlier questions indicated the 
roadmap that would be followed through the rest of the questionnaire. The combination 
of these two techniques allowed the respondent to complete the questionnaire more 
rapidly by not having to read inapplicable questions. The two middle groups of 
questions—core and general—are more complex and are discussed in further detail next. 

a. Core Questions 

The core questions request responses for each of the seven selected service 
categories and are further broken down into the following sub-groups: contract 
characteristics, acquisition management methods, project team approach and services 
acquisition leadership. The individual questions for each subject service category are 
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grouped together to facilitate ease of use for the respondent and so the respondent could 
think about and respond for one service category at a time. 

(1) Contract Characteristics. The questions in this sub-group 
pertain to each of the seven selected service categories. The survey again identifies the 
seven service categories considered by the questionnaire and provides a basic roadmap 
for this segment of the survey. The purpose of these questions is to gain insight into the 
dominant type of contract being used in the acquisition of services at the installation 
level. Answers to these questions will also provide information for each service category 
of whom—government or contractor—typically bears the risk associated with the 
contract and if there is enough competition for the services. The respondents are to base 
their selection on the dominant characteristic, for each service category, used in each 
fiscal year between 2002 and 2006. The characteristics examined in this section are 
competition (competitively bid or sole source); contract type (fixed-price or cost-type); 
and incentive/award (incentive fee or award fee or award term). 

(2) Acquisition Management Methods. The questions in this sub¬ 
group also pertain to each of the seven selected service categories. The purpose of these 
questions is to gain insight into the types of management structures being used at 
installations. The respondent was asked to reply based on the dominant services 
acquisition management method used at their respective installation for the various 
phases of the acquisition process. The acquisition process phases are acquisition 
planning, solicitation, source selection, and contract administration. For each of these 
phases the participant identified whether the phase was conducted at a regional, 
installation, or some other organizational level. An additional question in this sub-group 
led respondents to one of the next two questionnaire segments. The question asked the 
participant to identify whether a project team approach was typically used in the 
acquisition of the respective service category at the installation level. 

(3) Project Team Approach. The questions in this sub-group 
pertained only to those that identified a project team approach in the acquisition 
management method described above. If a project team approach was not selected in this 

section, SurveyMonkey logic allowed the participant to move onto the next sub-group of 
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questions. The respondent was asked to identify the billet of the project team leader, 
such as a Program/Project Manager or Contracting Officer. A related question presented 
to participants was who, by billet or organizational level, generated and approved 
changes to the requirements for service contracts. The purpose of these questions is to 
provide insight into what types of contracted services typically use a project team 
approach and to further the understanding of acquisition management methods used to 
manage services at the installation level. 

(4) Service Acquisition Leadership. The questions in this sub¬ 
group pertained to all seven service categories if the participant indicated that the project 
team approach was not dominantly used for that acquired service. For the applicable 
service categories, the participant was asked who, such as a Project/Program Manager or 
Contracting Officer, led the acquisition of that service at their installation. The 
participant was also asked who, by billet or organizational level, generated and approved 
changes to the requirements for service contracts. Again the purpose of these questions is 
to gain insight into what types of contracted services typically do not use a project team 
approach and to further the understanding of acquisition leadership methods used at the 
installation level. 

b. General Acquisition Management Methods Questions 

The general acquisition management methods questions call for a response 
pertaining to service acquisition methods in general for the respondent’s respective 
installation. The focus of this question group is a battery of 12 Likert-scale type 
statements. The respondent is asked to indicate their level of agreement of with each 
statement. Possible levels of agreement are: strongly disagree, disagree, neutral, agree 
and strongly agree. Other questions in this group ask the participant about the types of 
training received by contract/acquisition staff, contractor surveillance and the length of 
time contract/acquisition staff members serve in their billets. 

The last group of questions is presented to all respondents and offers the 
respondent the opportunity to make any general comments or provide other feedback 
regarding the topic of services acquisition. This is an important aspect of the survey to 
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facilitate the collection of additional information and data that may not have been 
captured within the body of the questionnaire. In addition the general comment block 
allowed the participant the opportunity to voice their concerns or make recommendations 
concerning the wording of individual questions, the length of time to complete the 
survey, or to provide other miscellaneous information. 

5. Data Analysis Plan 

The plan for analyzing data is to use various statistical tools to conduct an initial 
analysis of the data to help identify any trends or points of interest in the following 
categories: 1) intra-region, intra-military service, 2) intra-region, inter-military service, 
3) inter-region, intra-military service, 4) inter-region, inter-military service, and 5) inter¬ 
military service. 

C. IRB PROCESS 

The project team submitted a Protection of Human Subjects package to the Naval 
Postgraduate School’s IRB in accordance with the Naval Postgraduate School Instruction 
3900.4: Protection of Human Subjects (NAVPGSCOLINST 3900.4) for consideration. 
The package included a description of the survey (i.e. anonymous, web-based, and self- 
administered), informed consent information, a request for waiver of signed consent 
forms due to the nature of the survey, participant completion instructions, and a copy of 
all survey questions. The IRB approved our package to conduct the pilot test for the 
questionnaire. 

D. PILOT TEST 

The pilot test period was from late October to mid-November. It was conducted 
to ensure the individual questions and completion instructions were well written and 
easily understood by the respondent. Of the ten potential respondents contacted, six 
completed the web-based survey generating a response rate of 60 percent, which was 
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lower than expected. However, the empirical data and general comments provided by the 
respondents prove invaluable in generating overall survey reliability for use in follow-on 
research projects. 

E. REPORT OF RESULTS 

Chapter IV presents an initial analysis of the empirical data received from the five 
respondents to the pilot test. Chapter IV also presents numerous recommendations for 
improving the questionnaire to 1) facilitate follow-on research projects and 2) potentially 
increase the response rate to help provide a clear picture of the current state of acquisition 
management of services at the installation level. 
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IV. ANALYSIS OF PILOT DATA 


A. INTRODUCTION 

The pilot test obtained a response rate of 60 percent, or six completed surveys of 
the possible ten. Although six results are not statistically significant, they do provide 
some insight into the current state of acquisition management of services at the 
installation level within the DoD. This chapter presents an initial analysis of four service 
categories’—Professional, administrative and management support; Maintenance and 
repair of equipment; Data processing and telecommunications; and Transportation and 
travel—data collected during the survey’s pilot test. The other three service categories 
examined during the survey’s pilot test are not analyzed due to several factors which are 
discussed later in the chapter. In addition, an analysis of general acquisition management 
methods is presented. Finally, recommendations for improving individual questions and 
overall survey design are provided with the goal of facilitating an increased response rate 
during follow-on research projects. 

B. RESPONDENTS 

The six responses received were all from installations within CONUS. Sixty - 
seven percent of the responses came from Army installations, 17 percent from Marine 
Corps installations and 17 percent from Air Force installations. Eighty-three percent of 
responses were from installations west of the Mississippi River; 17 percent were from 
installations east of the Mississippi River. To protect respondent’s privacy and maintain 
confidentiality the identification of individual installation that responded is not 
provided—this information is maintained by the authors. 

C. SERVICE CATEGORIES NOT ANALYZED 

Three of the seven service categories examined during the survey’s pilot test did 
not generate enough raw data or substantial general comments to warrant statistical 
analysis. The three service categories not analyzed are Medical (FSC Q), Maintenance 
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and repair of real property (FSC Z) and Utilities and housekeeping (FSC S). Nearly all 
respondents skipped every question for each of these service categories. In the case of 
Maintenance and repair of real property and Utilities and housekeeping the general 
comments supplied indicated that these service categories were part of an overarching 
Base Operating Support (BOS) contract. There were no general comments provided 
which gave insight into how Medical services were acquired. 

D. SERVICE CATEGORIES ANALYZED 

The web-based, self-administered survey designed for this project was interested 
in seven of the 24 service categories as discussed in Chapter I. During the survey’s pilot 
test four of the seven service categories—Professional, administrative and management 
support (FSC R), Maintenance and repair of equipment (FSC J), Data processing and 
telecommunications (FSC D) and Transportation and travel (FSC V)—presented enough 
data for analysis and general conclusions regarding the current state of services 
acquisition management. 

I. Contract Characteristics 

The vast majority, 88 percent, of respondents indicated contracts within these four 
service categories were predominantly bid competitively during the five fiscal year 
period evaluated by the survey. The remaining 12 percent indicated that the contracts 
were sole sourced. Such a significant majority indicates that competition, at least for 
these service categories, is sought and adequate. 

Fixed-price contract was the dominant contract type used for acquiring these 
services. Seventy percent reported using fixed-price contracts indicating that contractors 
bore the preponderance of risk. Thirty percent utilized cost type contracts placing a 
higher level of risk on the government. 

Only one respondent indicated the use of incentives in the form of award fees. 
Award fees were only used in the acquisition of Maintenance and repair of equipment 
and Transportation and travel services. As reported it would seem that incentives could 

be better utilized in contracts for these services, possibly to drive contractor performance. 
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Yet the low use of incentives reported also shows less opportunities for abuse where 
incentives are awarded without the requisite superior performance level being met. 


2. Acquisition Management Methods 

The data show that the acquisition phases for each of the service categories 
continue to be conducted at the installation level. The only exceptions to this trend were: 
1) An Army installation reported the solicitation and source selection phases for 
Maintenance and repair of equipment were completed at the regional level, and 2) A 
Marine Corps’ installation indicated that they complete all acquisition phases at the 
regional level. The Marine Corps’ responses were consistent with their transition to 
regional contracting offices over the past few years. 

3. Project Team Approach 

Although data indicates a high percentage use of project teams for the acquisition 
of services, none of the respondents indicated a program manager leads the services 
acquisition team. The dominantly identified billet for leading the acquisition team was 
the contracting officer. 

Tables. Project Team Approach Used. 
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Team A 
typically 
the acqu 
services 
install 

’roject 
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Who, on-site (at your 
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service contracts? 
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No 

Contracting 
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Contracting 
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5 

1 
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1 

5 

Maintenance and repair 
of equipment 

3 

3 
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1 

4 

Data and 

telecommunications 

3 

3 

5 

1 

1 

5 

Transportation and travel 

4 

2 

3 
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3 

Total 

63% 

38% 

90% 

10% 

15% 

85% 


29 












































Despite the indieation that projeet teams are utilized in the aequisition of serviees, 
the projeet team eoneept is not employed in the same manner as in systems aequisition 
where a program manager leads the aequisition of the system and owns and writes the 
requirements for the system. The pilot survey results however indieate a possible 
diseonneet between the eontraeting offieer who leads the aequisition of serviees and the 
customer who owns and writes the requirements. 

E. GENERAL ACQUISITION MANAGEMENT METHODS 

This segment of the survey included four questions on general acquisition 
management characteristics at the installation level, regardless of service category. These 
questions were followed by a battery of 12 statements asking the respondent to indicate 
their level of agreement with the given statement. Possible levels of agreement were 
Strongly Disagree, Disagree, Neutral, Agree or Strongly Agree. The indicated levels of 
agreement to some of these statements pose several points of interest. 

From the four questions on general acquisition management characteristics one 
point of interest is the responses to the length of time Contracting Officer Representatives 
(COR) and Quality-assurance Evaluators (QAE) serve in their billets. Eight-three percent 
of COR and QAE personnel serve in their billets 2 or less years. The implied high 
turnover rate, especially the 33 percent who serve a year or less, can have negative 
impacts on the quality of contractor surveillance. 

Eigure 2. Time in Billet. 


■ less than 6 months 

■ 6 to 12 months 

□ 12 to 24 months 

■ 24 to 36 months 

□ over 36 months 
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High turnover rates are indicative of personnel having to take time to learn the 
new position and a decreased level of concentration while in the position due to a focus 
on follow-on assignments. These are some of the factors that lead to lower quality 
service contract oversight. 

In the last question 12 statements describe aspects of services acquisition, 11 of 
which should be the normal state of services acquisition at the installation level. First, 
four of these questions pertain to service acquisition personnel billets, manning levels, 
training and qualification. The results of the survey, shown in Figure 3, confirm what 
GAO has been reporting regarding the services acquisition workforce: that it was and 
still is undermanned, undertrained and under-qualified. All of the respondents disagreed, 
some strongly, that there are an adequate number of billets for services acquisition 
management at the installation level. They also overall disagreed with the following 
statement of acquisition management billets at the installation are adequately manned. It 
seems at 33 percent of installations services acquisition management staff members are 
adequately trained or qualified, but at best they are not above 50 percent. Although the 
results show nothing new, they clearly indicate we cannot expect things to improve until 
the situation changes. 

Figure 3. Personnel billets, manning, training and qualification. 
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Two of the 12 questions investigate the use of a lifecyele approaeh for managing 
acquired services. Fifty percent of the respondents disagreed that a lifecycle approach is 
used at their respective installation (Figure 4) for both routine and non-routine services. 
The lack of a lifecycle approach for routine and non-routine services has the potential to 
place the government at a higher level of risk due to improper planning for the various 
phases in a service’s lifecycle. 

Figure 4. Lifecycle approach. 
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Figure 5 depicts statements regarding Quality-assurance Evaluators including 
their training and their performance of duties as well as the level of oversight provided to 
monitor contractor performance, whether conducted by QAE or other personnel. The 
responses are spread across the spectrum, but are more negative than positive. 
Respondents overall agree that the training of QAE and submitting of reports but 
predominantly disagreed about affording a proper level of oversight to monitor contractor 
performance. This further confirms what GAO reported and shows that changes are still 
needed in this area as oversight is vital to ensuring adequate contractor performance. 

Eigure 5. Training and oversight. 
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III 


mr 


Strongly Disagree 
Disagree 


Neutral 


Agree 


Strongly 

Agree 


□ * Personnel responsible for the 
surveillance of contractors receive 
formal/documented training prior to 
assuming the duties as a 
Quality-assurance Evaluator (QAE). 

■ * Quality-assurance Evaluators (QAE) 
submit written reports regarding the 
performance/quality of work of 
contractors to the regional contracting 
officer/office for each service contract. 

□ * Services contracts are alforded the 
proper level of oversight to monitor 
contractor performance. 
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Finally, Figure 6 shows that the respondents predominantly agree that there is no 
discrepancy between requirements identification and Statements of Work/Objectives. 
Thus the cost increase is not due to miscommunication of requirements and objectives. 
Respondents also agreed that market research was conducted for the acquisition of 
services. 


Figured. Positive responses. 



Strongly Disagree Neutral Agree Strongly 
Disagree Agree 


□ * The entity that identifies the 

requirements in a service contract also 
writes the Statement of Work/Statement of 
Objectives (SOW/SOO) for the service 
contract. 


I * Market research is conducted for the 
acquisition of services. 


F. RECOMMENDATIONS TO IMPROVE SURVEY 

The completion of the pilot test not only gathered valuable data for analysis, it 
provided insight into how the survey questions were received by the respondents and 
whether the questions were applicable to the current state of services acquisition 
management. 

The first survey improvement recommendation is to provide more concise and 
clear instructions to participants. The current completion instructions do not clearly state 
that a team approach may be used to answer the survey questions. A team approach to 
providing answers may improve the snapshot of acquisition management of services for 
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that particular installation. The instructions still need to be clear that each installation 
should only submit one complete survey and to not allow individual team members to 
submit partially completed surveys. 

A second survey improvement recommendation is to decrease the number of 
survey questions presented at one time. The survey issued during the pilot test contains 
85 questions, one of which has a subset of 12 questions. Although Survey Monkey 
provides means to implement skip logic, 85 questions may be just too long. A means of 
shortening the survey would be to remove the three service categories—Medical; Utilities 
and housekeeping; and Maintenance and repair of real property—that did not generate 
enough data or information for analysis. 

Another survey improvement recommendation is to create a survey based solely 
on a battery of questions that can be answered using a Likert-type scale. During the pilot 
test of the current survey, all respondents supplied an answer for each of the 12 questions. 
Future additional questions should ask if manning levels and billet fill rates have declined 
and the level of decline over the past five years. This tactic of asking questions may be 
faster for the respondent to complete and still provide significant insight into the 
management of services acquisition. 

A fourth survey improvement recommendation is to add a question within each 
service category that requests quantitative data regarding the dollars obligated for the 
service category in each year for a range of fiscal years. The recommended means to 
accomplish this is to provide the participant with several dollar value ranges and require 
that they select one of the ranges by checking a box. An alternative way of capturing 
quantitative data is to provide a blank text box and require the respondent to input data. 
The recommended method allows the survey designer to choose and format the dollar 
value ranges; the second method requires the survey designer to clearly convey to the 
participant the format of the dollar value entry (e. g. use of $ or comma or whole dollars). 

Lastly, the survey should add a question asking each respondent to indicate the 
amount of time spent to complete the survey. The survey design options for this 
question—range or text box—are similar to those for dollars obligated discussed above. 
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V. SUMMARY, CONCLUSIONS AND RECOMMENDATIONS 


A. SUMMARY 

The goal of this project was to gain insight into the current state of acquisition 
management of services at the installation level and how to best collect empirical data in 
this area of study. Chapter II provided an overview of survey methodology and based on 
the advantages and disadvantages of various survey types, a web-based, self-administered 
survey was deemed the best method to collect the desired empirical data. Chapter III was 
a discussion of the various survey design aspects of the survey presented during the pilot 
test. Chapter IV analyzed the pilot test data for four of the seven service categories and 
made several recommendations to improve the survey to facilitate follow-on research 
projects. Here in Chapter V, the project’s conclusions and recommendations for further 
study are presented. 

B. CONCLUSIONS 

The most appropriate research method to collect the amount of desired data in this 
area of study is to conduct a web-based, self-administered questionnaire. The survey 
created for this project serves as a baseline for future research projects. Without a pilot 
test of a survey to test the questions in a real world environment, reliable and valid survey 
results cannot be expected from a larger sample of the installation population. By 
incorporating the recommended design changes to improve the survey, further data 
collection efforts will yield higher quality data and provide greater insight into the 
management of services acquisition at the installation level across DoD. 

The current state of services acquisition management at the installation level, as 
revealed by the initial analysis of survey pilot test data, demonstrates several of the key 
aspects causing increases in service contracts. Some of these aspects include deficit billet 
and manning levels which are further exacerbated by inadequate training and experience 
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of acquisition personnel. The lack of robust project team and lifecycle approaches in the 
acquisition management of services is another factor contributing to ineffective and 
inefficient management. 

C. RECOMMENDATIONS FOR FURTHER STUDY 

Acquisitions management, whether for a weapon system or a service, contains a 
broad range of topics deeming multiple opportunities for further study and potential 
management improvement recommendations. The authors present some areas for further 
study in the management of services acquisition area. 

In Chapter IV the discussion of which service categories were not analyzed 
identified that several service categories are sometimes grouped together under an 
overarching Base Operating Support (BOS) contract. Studies focusing on BOS contracts 
could examine the types of services typically bundled together in a BOS contract. 
Empirical data could be collected to identify trends in contract characteristics, which 
acquisition management methods are used, or if the military services handle this type of 
contract differently. 

Additional recommendations for further study focus on expansion. The first is to 
expand the geographical boundaries of the survey by issuing it to all CONUS and 
OCONUS installations for all of the military services. A worldwide survey would help 
identify trends between CONUS and OCONUS installations along with trends between 
the military services. 

Other expansion options include 1) conduct surveys to capture empirical data 
regarding the other 17 service categories, 2) expand the number of fiscal years considered 
to identify long-term trends, and 3) create a survey to gather quantitative data on service 
contracts administered under the Simplified Acquisition Process. 

One last recommendation for further study in the management of services 
acquisition is to examine other DoD agencies and field activities such as the Defense 
Logistics Agency (DLA), the Defense Finance and Accounting Service (DFAS) and the 
Defense Technical Information Center (DTIC). 
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